Diagnosis and treatment planning for patients can be significantly improved by comparing the patient's images with clinical images of other patients with similar anatomical and pathological characteristics, where the similarity is based on the understanding of the image content. There are two kinds of search: semantic search in patient records (=structured data) and similarity search on images using low-level image features.
The first requires semantically annotated images (i.e., labeled image regions) using a common vocabulary from a knowledge databases. This structured information is stored side-by-side with the images. For storing semantic information and background knowledge one often uses ontologies. The second search type uses similarity metrics based on pixel intensities.
With these two query methods the user can search the content of images, texts and other clinical values acquired at the hospital for similar medical cases.
Current systems show weak search capabilities. Therefore the user must remember a similar medical case and search by the patient's name.